Papers with self-speculative decoding
Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Speculative decoding uses a small draft model to generate a single input token, instead of sequentially generating tokens until completion. |
| Approach: | They propose a method that generates varying draft models adapted to the input context using simple rules. |
| Outcome: | The proposed method is competitive with the current SOTA for self-speculative decoding while being a truly plug-and-play method. |
Draft
& Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding. |
| Approach: | They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model. |
| Outcome: | The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint. |
Pre-Training Curriculum for Multi-Token Prediction in Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Multi-token prediction (MTP) is a pre-training objective for language models . prior work has shown that smaller language models struggle with the MTP objective . |
| Approach: | They propose a curriculum learning strategy that uses multiple prediction heads to predict the next tokens at each prediction step. |
| Outcome: | The proposed curriculum improves performance and output quality while retaining the benefits of self-speculative decoding. |
CLaSp: In-Context Layer Skip for Self-Speculative Decoding (2025.acl-long)
Copied to clipboard
Longze Chen, Renke Shan, Huiming Wang, Lu Wang, Ziqiang Liu, Run Luo, Jiawei Wang, Hamid Alinejad-Rokny, Min Yang
| Challenge: | Existing methods for drafting Large Language Models require additional modules to be trained, which can be challenging to implement and ensure compatibility across various LLMs. |
| Approach: | They propose an in-context layer-skipping strategy for self-speculative decoding that uses a plug-and-play mechanism to skip intermediate layers of the verify model to construct a compressed draft model. |
| Outcome: | The proposed method achieves a speedup of 1.3 1.7 on LLaMA3 series models without altering the original distribution of the generated text. |